Lazy Loading Support for Pydantic Settings Sources

Summary

This PR implements lazy loading for settings sources, deferring field value resolution until fields are accessed rather than fetching all values during initialization. This enables significant performance improvements for expensive operations such as API calls to cloud secret managers.

Solves #713

What Changed

• Added a new lazy_load parameter to GCP Secret Manager settings source (GCPSecretManagerSettingsSource) to opt into lazy loading.
• Implemented lazy loading support in the base class, ensuring all providers can benefit with minimal changes.
• Introduced the internal LazyMapping mechanism for deferring and caching per-field lookups.
• Updated tests to cover lazy loading behavior and environment-based sources.
  • Performed integration testing specifically for the GCP provider.

Problem

Currently, all settings sources eagerly fetch values for every field during Settings instantiation, even if those fields are never accessed. This is problematic for expensive operations:

• API calls to cloud secret managers (GCP Secret Manager, AWS Secrets Manager, Azure Key Vault)
• Large file reads from secrets directories
• Network roundtrips that could be avoided

Solution: LazyMapping

The implementation introduces a LazyMapping class, a dict-like mapping that:

• Defers field value resolution until keys are accessed via __getitem__()
• Caches computed values to avoid redundant operations
• Implements the Mapping ABC for compatibility with Pydantic's initialization

When lazy_load=True:

• Settings sources return an empty dict from __call__()
• A LazyMapping is stored on source._lazy_mapping
• Field values are only fetched when explicitly accessed

Test Coverage

• unit tests covering LazyMapping behavior and GCP Secret Manager

Note: Integration tests were performed for GCP Secret Manager. The fix is implemented at the PydanticBaseEnvSettingsSource class level, so all inheriting providers automatically support lazy loading. The parameter was only added for GCP Secret Manager, but extending it to new providers should be as simple as adding the parameter.

Why LazyMapping

Backward Compatibility

lazy_load defaults to False, preserving eager loading behavior.

Alternative Approaches Considered

I don’t think this is the most intuitive implementation, and I initially wanted something simpler. However, I've discussed some other options and nothing convinced me:

1. Lazy attribute access on Settings (__getattr__)

   • Idea: Fetch values only when you access them (e.g., settings.db_password)
   • Problem: Requires hacky code that intercepts all field access. Your IDE won't know what fields exist, autocomplete breaks, and it breaks every time Pydantic updates.

2. Separate LazySettings class

   • Idea: Have two different Settings classes—one eager, one lazy. Pick which to use.
   • Problem: Users have to decide at import time. Can't mix lazy and eager sources together.

3. Property-based field access

   • Idea: Turn each Settings field into a function/property that fetches on demand
   • Problem: Users would have to change how they define every single field in their code. Your IDE won't understand the types anymore.

4. Async initialization

   • Idea: Use async def __init__() to fetch values asynchronously
   • Problem: Would break existing code massively. Every Settings instantiation would need await. Too invasive.